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NOTE: To avoid abandonment of the application, the appHcani shali furnish to the USPTO, not later than 20 
, months from the priority date: (1) a copy of the intema^onai application, unless it has been previously 
' Communicated by the International Bureau or unless it was originally filed in the USPTO; and (2) the 

basic national fee (see 37 C.F,R. § 1,492(a}). The 30-month time limit may not be extended, 37 C.F.R, 

§ 1,495, 

WARNING: Where the items are those which can t>e submitted to complete the entry of the international 
application into the national phase are subsequent to 30 months from the priority date the 
application is still considered to t>e in the international state and if mailing procedures are utilized 
J to obtain a date the express mail procedure of 37 C.F.R ^ 7.70 must be used (since international 

application papers are not covered by an ordinary certificate of mailing — See 37 OF,R, § 1.8, 

NOTE: Documents and fees must be cleariy identified as a submission to enter the national state under 35 
U.S.C. §371 otherwise the submission wilt be considered as being made under 35 U.S.C, § 111, 37 
§ 1,494(f), 

L Applicant herewith submits to the United States Elected Office (EO/US) the following 
items under 35 U.S.C. § 371: 

a. [3 "^is express request to immediately begin national examination procedures 

(35 U.S.C. § 371(f)). 

b. □ The U.S. National Fee (35 U.S.C. § 371(c)(1)) and other fees (37 C.F.R. § 1.492) 

as indicated below: 
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CLAIMS 


(1) FOR 


(2) NUMBER 


(3) NUMBER 


(4) RATE 


(5) CALCULA- 


FEE 




FILED 


EXTRA 




TIONS 




TOTAL 












CLAIMS 












9 


9 "io- 


0 


X $18.00^ 


$ 0 




INDEPENDENT 












CLAIMS 












9 


9 -3 = 


6 


X $80.00 = 


480,00 




MULTIPLE DEPENDENT CLAIM(S) Of applicable) 


+ $270.00 




BASIC FEE** 


□ U.S. PTO WAS INTERNATIONAL PRELIMINARY EXAMINATION 






AUTHORITY 












Where an International preliminary examination fee as set forth 






in § 1.482 has been paid on the international application to the 






U.S. PTO: 












□ and the international preliminary examination report 






states that the criteria of noveUy, inventive step (non- 






obviousness) and industrial activity, as defined in PCT 






Article 33(1) to (4) have been satisfied for all the 






claims presented in the application entering the 






national stage (37 C.RR. § 1.4g2(a)(4)) .. 


$100.00 






□ and the above requirements are not met (37 C.F.R. 






§ 


1.492(a)(1)) 










as. PTO WAS NOT INTERNATIONAL PRELIMINARY 






EXAMINATION AUTHORITY 










Where no international preliminary examination fee as set forth 






in § 1.482 has been paid to the U.S. PTO, and payment of an 






international search fee as set forth in § 1.445(a)(2) to the U.S. 






PTO: 












□ has been paid (37 C.F.R, § 1.492(a)(2)) .. 


$710.00 






□ has not been paid (37 C.F.R. § 1.492(a)(3)) ..... $1000.00 






)C2 where a search report on the international application 






has been prepared by the European Patent Office or 






the Japanese Patent Office (37 C.F.R, 








§ 1.492(a)(5)) 






860.00 








Total of above Calculations 














1,340.00 


SMALL 


Reduction by 1/2 for filing by small entity, if applicable. Assertion 




ENTITY 


must be made, (note 37 C.F.R. § 1.27) 








Subtotal 










Total National Fee 


$ 












1 "^^n nn 




Fee for recording the enclosed assignment document $40.00 (37 






C.F.R. § 1.21(h)). (See Item 13 below). See attached "ASSIGNMENT 






COVER SHEET", 








TOTAL 


Total Fees enclosed 


$ 
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*See attached Preliminary Amendment Reducing the Number of Claims. 
P Attached is a Q check □ money order in the amount of $ 



S3 Authorization is hereby made to charge the amount of $ U34Q.00 

£3 to Deposit Account No. __5O-0510 

□ to Credit card as shown on the attached credit card infonnation authoriza- 
tion form PTO-2038. 
WAftNING: Credit card information should not be included on this form as it may become public. 

(3 Charge any additional fees required by this paper or credit any overpayment 
in the manner authorized above. 

A duplicate of this paper is attached. 

"*WARNING: "To avoid abandonment of the application the applicant shall furnish to the United States Patent 
and Trademark Office not fater than the expiration of 30 months from the priority date: * ' * (2) 
the basic national fee (see § 1.492(a)), The SO-month time limit may not be extended." 37 C.F.R. 
§ 7.495(b), 

WARNING: If the translation of the international application and/or the oath or declaration have not been 
submitted by the applicant within thirty (30) months from the priority date, such requirements may 
fc>e met within a time period set by the Office, 37 C.F.R, § 1.495(b)(2). The payment of the surcharge 
set forth in § 1.492(e) is required as a condition for accepting the oath or declaration later than 
thirty (30) months after the priority date. The payment of the processing fee set forth in § 1.492(f) 
is required for acceptance of an English translation later than thirty (30) months after the priority 
date. Failure to comply with these requirements will result in abandonment of the application. The 
provisions of § 1.136 apply to the period which is set Notice of Jan. 3, 1993, 1147 O.G. 29 to 
40. 

□ Assertion of Small Entity Status 

□ /^pficant hereby asserts status as a small entity under 37 C.F.R. § 1.27. 

NOTE: 37 C.F.R. § 1.27(c) deals with the assertion of small entity status, whether by a written specific 
declaration thereof or by payment as a small entity of the basic filing fee or the fee for the entry into 
the national phase as states: 

"(c) Assertion of small entity status. Any party (person, small business concern or nonprofit 
organization) should make a determination, pursuant to paragraph (f) of this section, of entitlement 
to be accorded smafi entity status based on the definitions set forth in paragraph (a) of this section, 
and must, in order to establish smaJI entity status for the purpose of paying small entity fees, actually 
make an assertion of entitlement to small entity status, in the manner set forth in paragraphs (c)(1) 
or (c)(3) of this section, in the application or patent in which such small entity fees are to be paid. 

(1) Assertion by writing. Small entity status may be established by a written assertion of entitlement 
to small entity status. A written assertion must: 

(i) Be cleariy Identifiable; 

(if) Be signed (see paragraph (c)(2) of this section); and 

(iii) Convey the concept of entitlement to small entity status, such as by stating that appdcant 
is a small entity, or that small entity status is entitled to be asserted for the application or patent. 
WhiJe no specific words or wording are required to assert small entity status, the intent to assert 
small entity status must be cleariy indicated tn order to comply with the assertion requirement. 

(2) Parties who can sign and file the written assertion. The written assertion can be signed by: 

(i) One of the parties identified in §§ 1.33(b) (e.g.. an attorney or agent registered with the Office), 
§§ 3.73(b) of this chapter notwithstanding, who can also file the written assertion; 

(ii) At least one of the individuals identified as an inventor (even though a 1.63 executed oath 
or declaration has not been submitted), notwithstanding §§ 1.33(b)(4), who can also file the 
written assertion pursuant to the exception under §§ 1.33(b) of this part; or 

(iii) An assignee of an undivided part interest, notwithstanding §§ 1.33(b)(3) and 3.73(b) of this 
chapter, but the partial assignee cannot file the assertion without resort to a party identified under 
§§ 1.33(b) of this part. 
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(3) Assertion by payment of the smafl entity basic filing or basic natiWAjee/il^k ^^aymeni ''by any ~ 
party, of the exact amount of one of the smafl entity basic filing fees set forth in §§ 1.16(aX (f), 
(Qh (h), or (}<), or one of the small entity basic national fees set forth in §§ 1.492(a)(1), (a)(2), (a)(3), 
(a)(4), or (a)(5), will be treated as a written assertion of entitlement to small entity status even if the 
I * type of basic filing or basic national fee is inadvertently selected in error. 

(i) If the Office accords small entity status based on payment of a sn7aW entity basic filing or basic 
national fee under paragraph (c)(3) of this section that is not applicable to that application, any 
balance of the small entity fee that is applicable to that application will be due along with the 
appropriate surcharge set forth in §§ 1.16(e), or §§ 1.16(1). 

(ii) The payment of any small entity fee other than those set forth in paragraph (c)(3) of this section 
t (whether in the exact fee amount or not) will not be treated as a written assertion of entitlement 

to small entity status and will not be sufficient to establish small entity status in an application 
or a patent." 

3. □ A copy of the International application as filed (35 U.S.C. § 371(c)(2)): 

NOTE: Section 1.495 (b) was amended to require that the basic national fee and a copy of the international 
application must be filed with the Office by 30 months from the priority date to avoid abandonment. 
"The international Bureau normally provides the copy of the international application to the Office in 
accordance with PCT Article 20. At the same time, the International Bureau notifies applicant of the 
communication to the Office, in accordance with PCT Rule 47, 1, that notice shall be accepted by all 
designated offices as conciusive evidence that the communication has duly taken place. Thus, if the 
applicant desires to enter the national stage^ the applicant normally need only check to be sure the 
notice from the Internationa! Bureau has been received and then pay the basic national fee by 30 months 
from the priority date." Notice of Jan. 7, 7993, 1147 O.G, 29 to 40, at 35-36. See item 14c below. 

a. □ is transmitted herewith. 

b. □ is not required, as the application was filed with the United States 

Receiving Office. 

c. □ has been transnrjitted 

i. □ by the International Bureau. 

Date of mailing of the application (from form PCT/1B/308): 



2C 



ii. □ by applicant on (Date) 

4. K A translation of the Intemationai application into the English language 
(35 U.S»C- § 371(c)(2)): 

a. K is transmitted herewith. 

b. □ is not required as the application was filed in English. 

c. □ was previously transmitted by applicant on (Date) 

d. Q wili follow. 
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6. Kl Amendments to the claims of the intemational application under PCT Article 19 
(35 U.S.C. § 371(c)(3)): 

NOTE: The Notice of January 7, 1993 points out that 37 C.ER, § 1, 495(a) was amended to clarify the existing 
and continuing practice that PCT Article 19 amendments must be submitted by 30 months from the 
priority date and this deadline may not be extended. The Notice further advises that: "The failure to 
do so will not result in loss of the subject matter of the PCT Article 19 amendments. Applicant may 
submit that subject matter in a preliminary amendment Wed under section 1.121. In many cases, filing 
an amendment under section 1.121 is preferable since grammatical or idiomatic errors may be 
■ corrected.'' 1147 O.G. 29-40, at 36. 

a. □ are transmitted herewith. 

b. □ have been transmitted 

i. □ by the Internationa! Bureau. 

Date of mailing of the amendment (from form PCT/1B/308): 



ii. □ by applicant on (Date) 

c. have not been transmitted as 

i. Kl applicant chose not to make amendments under PCT Article 19. 

Date of mailing of Search Report (from form PCT/iSA/21O0: 
4/18/00 

ii. □ the time limit for the submission of amendments has not yet 

expired. The amendments or a statement that amendments have 
not been made will be transmitted before the expiration of the time 
limit under PCT Rule 46.1. 

6. n A translation of the amendments to the claims under PCT Article 19 

(38 U.S.C. § 371(c)(3)): 

a. □ is transmitted herewith. 

b. □ is not required as the amendments were made in the English language, 

c. □ has not been transmitted for reasons indicated at point 5(c) above. 

7. E3 A copy of the international examination report (PCT/iPEA/409) 

Si is transmitted herewith. 

□ Is not required as the application was filed with the United States 
Receiving Office. 

8. □ Annex(es) to the intemational preliminary examination report 

a. □ is/are transmitted herewith. 

b. □ is/are not required as the application was filed with the United States 

Receiving Office. 

9. □ A translation of the annexes to the international preliminary examination report 

a. □ is transmitted herewith, 

b. □ is not required as the annexes are in the English language. 
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10. OCX An oath or declaration of the inventor (35 U.S.C. § 371(c)(4)) complying with " ^^^^ 



35 U.S.C. § 115 

a. □ was previously submitted by applicant on (Date) 

! ' b. □ is submitted herewith, and such oath or declaration 
i. □ is attached to the application. 

il. □ identifies the appJication and any amendments under PCT Article 
19 that were transmitted as stated in points 3(b) or 3(c) and 5(b); 
and states that they were reviewed by the inventor as required by 
i 37 C.F.R. § 1.70. 

c. H will follow. 

II. Other document(s) or information included: 

11- □ An International Search Report (PCT/ISA/210) or Declaration under 
PCT Article 17(2)(a): 

a. H is transmitted herewith. 

b. □ has been transmitted by the Internationa! Bureau, 

Date of mailing (from form PCT/iB/308): 

c. □ is not required, as the application was searched by the United States 

International Searching Authority. 

d. □ will be transmitted promptly upon request. 

e. □ has been submitted by applicant on (Date) 

12. E An Infomnation Disclosure Statement under 37 C.F.R. §§ 1.97 and 1.98: 

a. □ is transmitted herewith. 
Also transmitted herewith is/are: 

□ Form PTO-1449 (PTO/SB/08A and 08B). 

□ Copies of citations listed. 

b. 0 win be transmitted within THREE MONTHS of the date of submission 

of requirements under 35 U.S.C. § 371(c). 

c. □ was previously submitted by applicant on (Date) 

13. ^ An assignment document is transmitted herewith for recording. 

A separate □ "COVER SHEET FOR ASSIGNMENT (DOCUMENT) ACCOMPA- 
NYING NEW PATENT APPLICATION" or □ FORM PTO 1595 is also attached. 

TO Follow: International Business Machines CorporatioD^ Armonk, New York 
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14. H Additional documents: 

; ♦ a. □ Copy of request (PCT/RO/101) 

b. S International Publication No. W O 00 / 45 3 3 5 

i. (3 Specification, claims and drawing 

ii. □ Front page only 

c. □ Preliminary amendment (37 C.F,R, § 1,121) 

d. □ Other 



15. H The above checked items are being transmitted 

a. ED before 30 months from any claimed priority date. 

b. □ after 30 months. 

16. □ Certain requirements under 35 U.S.C. § 371 were previously submitted by the 

applicant on , namely: 



AUTHORIZATION TO CHARGE ADDITIONAL FEES 

WARNING: Accurately count claims, especially multiple dependant claims, to avoid unexpected high charges 
if extra claims are authorized, 

NOTE: "A written request may be submitted in an application that is an authon'zation to treat any concurrent 
or future reply, requiring a petOson for an extension of time under this paragraph for its timely submission, 
as incorporating a petition for extension of time for the appropriate length of time. An authorization to 
charge all required fees, fees under § 1.17, or all required extension of time fees will be treated as 
a constructive petition for an extension of time in any concurrent or future reply requiring a petition 
for an extension of time under this paragraph for its timely submission. Submission of the fee set forth 
in §1.1 7(a) will also be treated as a constructive petition for an extension of time in any concurrent 
reply requiring a petition for an extension of time under this paragraph for its timely submission." 37 
C.F.R J 1.136(a)(3). 

NOTE: "Amounts of twenty- five dollars or less will not be returned unless specifically requested within a 
reasonable time, nor will the payer be notified of such amounts; amounts over twenty-five dollars may 
be returned by check or, if requested, by credit to a deposit account'* 37 C.F.R § 1 26(a). 

H Please charge, in the manner authorized above, the following additional fees that 
may be required by this paper and during the entire pendency of this application: 

a 37 C.F.R. § 1.492(a)(1). (2), (3), and (4) (filing fees) 

WARNING: Because failure to pay the national fee within 30 months without extension (37 C.F.R. § 1, 495(b)(2)) 
results in abandonment of the application, it would be best to always check the above box. 
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a 37 C.F.R, § 1.492(b), (c) and (d) (preser^fr^B Sgr^Sr^^^© 2 6 JUL 
Because additional fees for excess or multiple dependent claims not paid on filing or on later presentation 
must only be paid or these claims cancelled by amendment prior to the expiration of the time period 
^ set for response by the PTO in any notice of fee deficiency (37 C.F.R § 1A92(d)), it might be best 
. f not to authorize the PTO to charge additional claim fees, except possible wfien dealing with amendments 
after final action. 



□ 



NOTE: 



NOTE: 



37 C.F.R, § 1.17 (application processing fees) 

37 C.F.R. § 1.17(a)(1H5) (extension fees pursuant to § 1.136(a). 

37 C.F.R. § 1.18 (issue fee at or before mailing of Notice of Allowance 
pursuant to 37 C.F.R. § 1.311(b)) 

Where an authorization to charge the issue fee to a deposit account has been tiled before the mailing 
of a Notice of Allowance, the issue fee will be automatically charged to the deposit account at the time 
of mailing the notice of allowance. 37 C,F,R, § 1,3n(b}. 

37 C.F.R ^ 1.2a(b) requires "Notification of any change in loss of entitlement to small entity status must 
be filed in the application . . . prior to paying, or at the time of paying , . . issue fee, " From the wording 
of 37 C.F.R § 1, 28(b): (a) notification of change of status must be made even if the fee is paid as "other 
than a small entity" and (b) no notification is required if the change is to another small entity. 

m 37 C.F.R. § 1.492(e) and (f) (surcharge fees for filing the declaration 
and/or filing an English translation of an International Application later 
than 30 months after the prioritySate). 



Reg. No.:24^522 

Tel. No.: C03 ) 259-1800 

Customer No.: 2512 




VATURE OF PRAfctmON^R 

Clarence A. Green 



{type or print name of practitioner) 
PERMAN & GREEN, LLP 



P.O. Address 
425 Post Road, Fairfield, CT 06430 USA 
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SPECIFICATION 



METHOD AND DEVICE FOR DESCRIBING VIDEO CONTENTS 



Field of the Invention 

The present invention relates to a method and an apparatus 
for describing contents of motion picture for the sake of 
efficient search based on contents of motion picture of a 
computer system capable of processing motion picture, in 
particular, video, DVD and so on. 



Background art 

While motion picture such as video data is easy for a human 
being to understand, there was a difficulty for a computer 
to manage it. Namely, it is difficult to find the meaning 
that the contents have from an original video data itself, 
and it has not been possible to date to accurately represent 
the meaning of video data even with advanced image 
processing technology. 
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While management of motion picture such as video data with a 
computer was conventionally based on annotations made by an 
administrator in advance, the method of using annotations 
lacks consistency in annotations among administrators, and 
besides, complicated processing for video data in large 
quantities is a big problem from now on. 

As a candidate for the solution, it is considered promising 
to describe metacontents from video data with an 
intermediate result of image processing and knowledge 
registrable in advance that the contents have. 

However, though it is possible to use plural image features 
for specific contents so as to describe a description method 
or a search engine for specific search or management, 
general versatility is lost and it does not contribute to 
proliferation of video search. 

Therefore, a description method with general versatility 
taking advantage of image features is desired for 
description of video data, and the activities for 
standardization were started by ISO (International 
Organization for Standardization) as MPEG-7. 
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MPEG (Moving Picture Experts Group) is an organization 
promoting standardization of encoding method for storing 
color motion picture, and MPEG-1, MPEG- 2 and MPEG- 4 have 
been standardized so far. 

Since MPEG- 7 has no rule for a method of image processing 
and is beyond the scope of any standards, not only automatic 
processing but also manual data input is allowed. 
However, it will only make data input more complicated to 
demand meaning of originally unextractable scene from video 
data or registration of data difficult to detect from video 
data. So far, there have been many examples of representing 
frame sequence of video as structured. For instance, Abe's 
method ("Method for Searching Moving Picture with Change of 
State over Time as Key", Abe, Sotomura, Shingakuron, 
pp. 512-519, 1992 (conventional example 1)) describes a 
dynamic change of state so that the time intervals to be 
searched in video search may not be fixed. 

In Abe * s method (conventional example 1), however, since 
information of state description covers the entire frames, a 
drawback is that search time is in proportion to length of 
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the video used for search. Also, since an object is 
represented by the center of gravity in an image, it is 
substantially different from the present invention taking 
advantage of changes in an object's shape. 

While the method described in conventional example 2, ("An 
Automatic Video Parser for TV Soccer Games," Y.Gong, 
C.H-Chuan, L.T.Sin, ACCV '95, pp. 509-513, Nov., 1995) is 
trying to use information of positions and movement of the 
players, the positions are classified code of positions 
where the field is roughly divided into nine, and movement 
is of a very short period (several frames) so that event 
extraction is performed regarding classified code of 
positions and motion vector in a short period as events. 
In conventional example 2, however, a drawback is that an 
event to be extracted and description are inseparable and 
besides, extractable events become a very limited set. 
The method described in conventional example 3, ("Integrated 
Image and Speech Analysis for Content Based Video Indexing, " 
Y-L. Chang, W.Zeng, I.Kamel, R.Alonso, ICMCS '96, pp. 306-313, 
1996) adopts a limited approach of tracking positions of the 
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ball and goal posts on the screen and considering only their 
positional relationship so as to extract time intervals of 
close distance as exciting scenes. 

The method described in conventional example 4, Analysis 
and Presentation of Soccer Highlights from Video," D.Yow, 
B.L.Yeo, M.Yeung, B.Liu, ACCV »95, pp. 499-502, 1995) 
performs shot extraction covering American football, and 
identifies events such as touchdown by a key word in each 
shot by speech recognition and line pattern extraction in 
the screen using image processing. 

However, neither conventional example 3 nor 4 has a concept 
such as a player and his movement. 

On the other hand, while there is also conventional example 
5, ("A Suggestion of a Method for Image Search by Inquiries 
Using Words of Movement," Miyamori , Kasuya, Tominaga, Image 
media processing symposium ^96, 1-8, 13, 1996) as a method 
for representation which cuts an object from video and is 
based on lifetime and an object position, it has neither a 
concept of reference plane nor general versatility. 
In addition, conventional example 6, ("A Suggestion of a 
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Method for Image Contents Search Using Description of Short 
Time Movement in a Scene," Miyamori , Maeda, Echigo, Nakano, 
lisaku, MIRU-98, 1-75, 1998) also describes an object with 
description of short time movement as a unit, but it lacks 
expandability since it does not simultaneously adopt 
description which represents a spatio-temporal trajectory 
and is a representation method dependent on specific 
contents . 

Conventional example 7, ("On the Simultaneous Interpretation 
of Real World Image Sequences and Their Natural Language 
Description: The System SOCCER," E.Andre, G.Herzog, T.Rist, 
Proc. 8th ECAI, pp. 449-454, 1988) is a system with scene 
description and interaction among objects as its meta data. 
However, the purpose of the system of conventional example 7 
is conversion of medium from image to speech, namely a 
system for automatically generating a narration, so it does 
not store the created meta data and unlike the present 
invention, it does not have a data structure suitable for 
search of contents. 

Conventional example 8, ("Automatic Classification of Tennis 
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Video for High-level Content -based Retrieval," G.Sudhir, 

J, CM. Lee, A.K.Jain, Proc . CAIVD-98, pp. 81-90, 1997) covers 

tennis matches, so description of interaction among objects 

is limited to simple movement and position information. 

The present invention is limited, in terms of its 

descriptive contents, to processing results based on 

"feature colors," "texture," "shape" and "movement." 

In video, a subject of attention is different depending on 

its contents. Therefore, it is necessary to predefine a 

subject object depending on the contents. 

An object defined here consists of a lump region appearing 
in image, and it is possible to extract its color, texture, 
shape and movement . 

It is a property of this object region that can be extracted 
from video, and it is difficult to give meaning to its 
contents . 

Accordingly, a technique of description based on 
relationship between a single object and plural objects is 
proposed, and knowledge dependent on contents registrable in 
advance and description of objects are associated, and thus. 
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search based on an object of a meaningful scene in video 
becomes possible. 

Since description of the entire frames of video data will 
result in storing of redundant information in large 
quantities, description for efficiently representing video 
contents with a little data volume is important. 
The present invention is a proposal of a description method 
effective for interpretation based on video contents. The 
description method of the present invention is effective not 
only for search of an object or a scene but also for 
applications such as reuse of an object and summarization of 
contents , 

An object of the present invention is to provide a 
description method for efficiently representing contents of 
motion picture with a little data volume. 

Another object of the present invention is to propose a 
description method effective for interpretation based on 
contents of motion picture. 

A further object of the present invention is to provide a 
description method not only for search of an object or a 
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scene but also capable of applications such as reuse of an 
object and summarization of contents. 

Summary of the Invention 

The present invention proposes a method for description in 
terms of relationship between a single object and plural 
objects based on data extractable as an image feature. 
Namely, the present invention (1) represents a trajectory of 
how each object has moved over time by using reference plane 
representing position information of each object, (2) sets a 
description unit based on a type of action of each object by 
using changes in shape of each object, (3) has actions of 
each object represented as each behavioral section, and (4) 
comprises a description facility capable of reading and 
interpreting definition of an object dependent on video 
contents, definition of classes of actions, and definition 
of interpretation of a scene by interaction of plural 
objects , 

Brief Description of the Drawings 
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Fig. 1 is a diagram showing cutting of video data on a 
computer to which the present invention is applied. 
Fig. 2 is a diagram showing that position information and 
actions are assigned to a behavioral section of each object 
to which the present invention is applied. 

Fig. 3 is a diagram showing a trajectory on reference plane 
of each object to which the present invention is applied. 
Fig. 4 is a diagram showing an outline of a processing flow 
of video data on a computer to which the present invention 
is applied along with its main components. 

Fig. 5 is a diagram showing a search screen of the present 
invention . 

Fig, 6 is a diagram showing a search result of the present 
invention , 

Preferred Embodiments 

As main features of an object in image, there are "position" 
of an object, its "moving speed" and "trajectory." 
However, in spite of conventional use of "position" 
information, position in description of video is mostly mere 
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implicit use of an image plane, and there is no method for 
using a different ground from the image plane depending on 
the contents . 

In addition, an image is originally a three-dimensional 
scene projected on a two-dimensional plane, and there are 
cases where it is more convenient to take an object position 
not on an image plane but on a plane on which the object 
exists , 

In general, sports images just fall under the cases where it 
is more convenient to take an object position not on an 
image plane but on a plane on which the object exists. 
While a plane equivalent as world coordinates system is 
generally used for description of an object of the real 
world, it is different from video in the subject of 
description, purpose and in that it always takes the ground 
for a part of the real world. 

On the other hand, since an image lacks information of a 
depth direction, it may be better to project consecutive 
frames of video on a common image plane. 

From the above consideration, it is necessary to predefine 
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for each of the contents to be searched the ground for 
determining an object position which is called reference 
plane and described as follows by using "zone description" 
and a "camera model." 

This description method allows effective description for the 
contents to which a geometrical relationship among objects 
is important. 

Descriptions of reference plane, zone description and camera 
spec. are shown in attached Table 1. However, the 

description of camera spec, is not essential to the present 
invention but optional. 

Next, representation method for each object is explained. 
An object moves on reference plane and performs meaningful 
behavior/actions in it. 

For the contents in which behavior of an object may be a 
major factor in hitting the search, specifically the 
following description method is very effective. 
A representation unit of an object is decomposed on the 
basis of behavior, and initiating and terminating frames 
representing its behavioral section and a trajectory between 
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them are described so as to reproduce an object position in 
an arbitrary frame , 

While nothing represented changes in shape of each object in 
the background art, the present invention allows meaning of 
an object of a description unit to be preserved by giving 
meaning to changes in shape as behavior of an object due to 
such changes in shape. 

This description is represented as Action (or described as 
Act) shown in attached Table 2, 

An example of description using this Action is as follows. 

Act("Kick", lO(frame), 240(frame), 3, 3, (120, 180, 0) (150, 
195, 180) (180, 223, 230)) 

By this description of Action, a representation of "An 
object (who) did the behavior indicated by action ID (what) 
on the space represented by a trajectory (where) in a time 
interval (when)" becomes possible. 

Here, collinear approximation is used as a method for 
representing a trajectory. Data of a trajectory is 
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collinearly approximated, and the number of nodes, 
approximated coordinates on reference plane of each node and 
the time in Action are described. Consequently, if a 
certain time is specified, coordinate value of an object at 
the time can be uniquely determined. 

Fig. 1 is a diagram conceptually representing description 
based on reference plane and a behavioral section of an 
object and description and data flow based on plural objects 
in soccer contents. 

First, a video object based on the region is cut from video 
sequence, and lifetime of the object is acquired by tracing 
each object in the direction of time (131). Next, actions 
of the object are classified based on silhouette of the 
object which represents changes in shape of each object. 
Description of objects is performed for each of these 
behavioral sections, which is Action description (132) . 
At this time, spatial movement of the object is represented 
as its trajectory by using reference plane. Objects present 
in the entire frames of video have Action description 
independently, and lAction (described later) is defined from 
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plural objects. 

For simplicity / a trajectory in Act is described here 
limiting it only to that of two points. In addition, a 
trajectory of a ball is used as to spatial representation of 
lAct (described later) . And in this case, information of a 
ball is described with a player's touch on a ball (kicking 
or receiving a ball) as a unit, regarding a touch between 
different players as a pass and a consecutive touches by the 
same player as a dribble. 

If Fig, 1 is seen further in detail, a region based on 
color, texture and movement is cut from video sequence 101 
and 102 (131) . While this process is basically automated, 
it is also possible to modify it in each field by a tool for 
correcting regions extracted erroneously and oversplit 
regions . 

Hereafter, the split moving regions are handled as video 
objects. Also, an object ID can be inserted at this time. 
Tracing of a region between neighboring fields is 
automatically determined by a size of the region where split 
regions are overlapping. A ball, as a special object, is 
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not currently cut, and manual input is performed with a tool 
for inputting a position of a ball on an image. Ill in Fig. 
1 shows a conceptual diagram of the above data. 
Next, a camera movement parameter is restored by tracing in 
each field a corner or a feature region on a still object in 
the background . 

Position of a video object is represented on an image plane, 
which requires correction according to movement of a camera. 
Therefore, a virtual plane is assumed and the video of which 
a movement parameter is restored is projected on an image 
plane. Thus, data equivalent to restoring an object 

position in video which is input from a single camera is 
acquired. Moreover, since the camera is set to have an 
angle of depression against the ground, it can be handled 
not as a distance on an image plane but as a real distance 
by projecting the object position on the plane of the ground 
looked down on from the sky as reference plane. This is 
represented in Space 123 in Fig. 1 as a conceptual diagram 
which is continuous with time. 

While the cut object is a video changing shape over time, in 
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this example, a silhouette disregarding internal color 
information is used in order to pay attention to changes in 
shape. Since changes in silhouette show specific changes 
according to actions, consecutive silhouettes of plural 
pre -instructed action patterns are unfolded in unique space 
so as to acquire changes unique to actions from high order 
eigenvalue . 

Next, when input patterns are unfolded in the same unique 
space, the movement patterns are identified by seeking which 
instructed patterns they are closest to. This process 
requires a changing point in actions in a series of object 
actions to be acquired. While it is possible to acquire the 
changing point from the unique space, it is currently 
entered manually. 

A series of objects can be described by plural actions, and 
just by entering an action ID and a frame number at a 
changing point in actions, it can be interpreted that all 
the objects in between were performing the same movements. 
As above, while the data transformation described in Fig. i 
has processing which partly needs manual input support, it 
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is possible to generate data in a process which is expected 
to be automated in the future . 

Fig. 2 is a diagram showing in detail the same drawing as 
TIME (124) in Fig. 1, but the subject video data is 
different . 

Horizontal axis 292 in Fig. 2 represents time, and vertical 
axis 291 represents an object ID. 

An object is described making the changing points of time of 
silhouettes 201 and 202 in Fig, 2 boundaries of description 
with an identified action ID as a minimum unit. The time 
intervals and object positions at the start /end of an action 
ID are described. Also, in order to trace an object 
trajectory, positions at plural points in a time interval 
are described. Accordingly, object positions can be 

approximated in all the frames. 

200 in Fig. 2 is a conceptual diagram of objects according 
to the time intervals where Objs. 1-6 (203-208) represent 
the players, (A) and (B) represent the teams, and Obj , X 
(209) represents the ball. 

Ball 209 is described as an object without an action ID. 
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For instance, Obj , 2 (A) is a player of team A and it runs 
during time interval 214, stays in time interval 224, kicks 
a ball in time interval 234, runs in time interval 244, 
kicks a ball in time interval 254, and runs in time interval 
264. These actions are recorded with starting time, ending 
time and an object position (Px) . 

Fig. 3 shows movement of major players in the soccer scene 
for about 20 seconds in a solid line and movement of a ball 
in a dotted line. While Fig. 3 is a diagram showing in 
detail the same drawing as Space (123) in Fig. 1, but the 
subject video data is different. 

These are the cases where the objects were extracted from 
the actual soccer scene and then the movement of each object 
on the field was reproduced by automatically extracting 
movement parameters of the camera. In the case of Fig. 3, a 
soccer field is set as reference plane. 

The ball object (dotted line) is kicked first from position 
312 and reaches position 314 near goal 340 after being 
kicked several times. 

It shows that a player (solid line) has moved from position 



19 



JA998222 



322 to position 324 and another player has started from 
position 312 and moved to position 314, 

In this way, a trajectory of each object in the video data 
on reference plane can be traced. 

Next, while in general, plural objects simultaneously exist 
in an image and have different lifetimes and meanings 
respectively, it becomes possible to give meaning to a scene 
consisting of behavior of plural objects. 

This is described as Interactive Action (lAction) . lAction 
(or described as lAct) is totally dependent on contents, and 
a different definition for each content or for each video 
database administrator may be used. 

However, in order to facilitate consistency of description 
of lAction and its application to a search engine, it is 
prescribed in this specification that lAction is defined by 
logical operation with another lAction and plural Actions. 
Description of lAction is shown in attached Table 3 . 
An example of description using this lAction is as follows. 

lActC'Pass", 20 (frame), 35 (frame) , 2, 1, 2, Trajectory, 2, 
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(120, 180, 0) (160, 230, 15)) 

The above description of lAction is a description of "when, 
where, who did what." The differences from the above Action 
are that its subject is plural objects and that the location 
is specified in a voluntary manner. 

In the example of soccer mentioned later, a trajectory of a 
ball is used as spatial representation of an event. 
As an example, descriptions of a "Pass" to which meaning is 
given by two objects and a "Through Pass" represented by AND 
of other objects are explained. 

Here, it is assumed that the database described by the above 
Act and lAct actually exists. It is defined for 

convenience' sake that a through pass is "a pass made 
between players A and B, and the ball went through between 
players C and D of the defense side who were there at the 
time the pass was made." 

In the present invention, it is defined as to determine the 
following. 1. There was an lAct ("pass") between certain 
two players. 2. There were two other players of the defense 
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side in that time interval (there are two Acts sharing a 
time interval with 1, and the object which performed the two 
Acts belongs to a different team from the objects of 1) , 3, 
The trajectory of the pass went through between the two 
players (in the shared time interval, the trajectory of the 
ball intersects the line linking the two players of the 
defense side) . 

Here, the lAct statement is described in attached Table 4. 
An example of definition of an lAct through pass is shown in 
attached Table 5. 

As an example of application of the present invention, 
description in a soccer game is shown. In soccer, a type of 
play defined from a single object and a type of actions 
defined by relationship of plural objects are used as 
description items. Moreover, in soccer, description of a 
ball object is used as a special object. 

Description of a ball itself can be represented likewise, 
namely an action ID is omitted in the Action description of 
the above object and the object ID is a ball. 

The following description defines lAction in soccer 
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contents, and description using Action and lAction is also 
possible to other contents. 

It can be considered a useful description method for the 
contents such as the present invention in which 
representation of behavior of an object and positional 
relationship on reference plane is effective, since there 
are a number of examples of application such as sports other 
than soccer, measurement of traffic and video surveillance, 
etc , 

Here, a ball object is described as follows, 

ball ( StartTime, EndTime, number of Nodes, Trajectory ) 

There are the following types of actions of a single object 
used in soccer. 

Activity = { lie, sit, fall_down, raise_hand, dive, 
hand__throw, throw_in, jump, stay, walk, run, sliding, kick, 
overhead_kick } 
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Namely, lie, sit, fall_down, raise__hand, dive, hand__throw, 
throw_in, jump, stay, walk, run, sliding, kick, 
overhead_kick, and so on can be considered in general. 
As examples of actions, a pass (lines 1 to 26 in Table 6) , a 
long pass (lines 28 to 39 in Table 6) , a feed pass (lines 1 
to 14 in Table 7) , a cross pass (lines 16 to 33 in Table 7) , 
a gain pass (lines 1 to 18 in Table 8), centering (lines 20 
to 38 in Table 8), and a wall pass (lines 1 to 36 in Table 
9) are described. 

As definition of a through pass as an example of lAct was 

already explained in detail referring to Table 5, detailed 

explanation of these examples is not given here. 

In addition, the groups of auxiliary functions used in lAct 

are shown in attached Tables 10 and 11, See the groups of 

auxiliary functions for the meanings of the functions used 

in the above examples (see attached Tables 6 to 9) . 

Fig. 4 shows the steps necessary for searching video data 

and related data groups (see attached Tables 10 and 11) . 

Video data 4 01 is processed by image processing 410 and the 

results are stored as data groups of 420. For instance, 
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region map 422 is acquired from a region cutting process, an 
object trajectory ID (424) from identification of an object, 
an action ID (426) from classification of actions, camera 
parameter 428 from camera action restoring process, etc. 
However, these processes are not only completely automated 
processes but also the cases of formatting data with manual 
support after generation and manually inputting as data 
directly from video data 401. 

From these data groups of 42 0, Act 43 0, namely video 
description according to the objects is obtained. 
Also, from the definition obtained in advance by selecting 
video data 401, description of reference plane (Refplane) 
442 for video data 401 is obtained. 

Furthermore, by selecting video data 4 01, scene description 
lAct 456 describing meaning given to a scene consisting of 
behavior of plural objects is obtained, which is predefined 
for interpreting video data 401, 

As an application of this description, to search video data 
401, a user will enter user keyword 470 to search engine 
460. Search engine 460 interprets user keyword 470 and 
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returns a time interval of corresponding video data 4 01 from 
object description 430 and scene description 456 to display- 
on video to the user. 

At this time, scene description 456 returns a corresponding 
time interval by processing Refplane 422 and Act 430. 
Moreover, a user is allowed a description equivalent to 
scene description lAct 456 for search engine 460, and 
Refplane 422 and Act 430 are processed for a user-defined 
scene description so that a time interval of a user-defined 
scene is returned and displayed on video to the user. 
Fig. 5 shows a screen for searching video. 

It is possible for a user to search a desired scene by 
selecting a necessary item in search screen 500 and starting 
search. 

In the case of a soccer video explained so far, it is 
possible to perform specification of players 510, 
specification of time 520, specification of location (place) 
530, and specification of action 540. 

As to specification of players, it is thinkable to specify 
them by team name, individual name or position. 
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In the case of specification of action, it is possible to 
use the actions defined in the above-mentioned Action and 
lAction. For instance, it is possible to use Actions such 
as lie, sit, fall__down, raise_hand, dive, hand_throw, 
throw_in, jump, stay, walk, run, sliding, kick or 
overhead__kick, or lActions such as a pass, a through pass or 
a centering. Moreover, a user may also newly define a 
scene . 

Fig. 6 shows screen 6 00 of a search result of a scene in the 
case of specifying a through pass for an action in Fig. 5. 
For instance, the search result in this case is one, and 
image 610 at the start of the scene is displayed. 
Generally, a desired scene is replayed by clicking this 
image 610 of the search result. 

Advantages of the Invention 

It is possible, by adopting the above-mentioned organization 
of the present invention, to provide a description method 
for effectively representing contents of motion picture with 
a small data volume. 



27 



JA998222 

Also, a description method effective for interpretation 
based on contents of motion picture becomes possible by 
adopting the organization of the present invention. 
Furthermore, it becomes possible, by adopting the 
organization of the present invention, to provide a 
description method of motion picture capable of applications 
such as reuse of an object and summarization of contents in 
addition to search for an object or a scene. 

As a summary, other embodiments are described below. 

A description means for contents of motion picture, the 
means comprising of: 

(a) means for setting reference plane; 

(b) means for describing each object on the motion picture 
by position on the reference plane and predefined type of 
actions; and 

(c) means for describing each scene by using the means for 
describing each object. 
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A search means for contents of motion picture, the means 
comprising of: 

(a) means for setting reference plane; 

(b) means for describing each object on the motion picture 
by position on the reference plane and predefined type of 
actions ; 

(c) means for describing each scene by using the means for 
describing each object; and 

(d) means for searching motion picture by using the means 
for describing each object or the means for describing each 
scene * 

A description method for motion picture, the method 
comprising the steps of; 

(a) determining reference plane which represents information 
of object positions included in the motion picture; 

(b) representing changes over time of each object on the 
reference plane as a trajectory; 

(c) setting a description unit based on predefined type of 
actions of each object by using changes in shape of each 



29 



JA998222 

object so as to assign actions of each object as each 
behavioral section; and 

(d) defining each scene by plural objects. 

A search method for motion picture, the method comprising 
the steps of : 

(a) setting reference plane which represents information of 
object positions included in the motion picture; 

(b) representing changes over time of each object on the 
reference plane as a trajectory; 

(c) setting a description unit based on predefined type of 
actions of each object by using changes in shape of each 
object so as to assign actions of each object as each 
behavioral section; 

(d) defining each scene by plural objects; and 

(e) searching a specific scene by using the actions of each 
object or the scene. 

A description method for motion picture, the method 
comprising the steps of: 
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(a) determining reference plane from the motion picture; 

(b) cutting a region map, an object trajectory ID, an action 
ID and a camera parameter from the motion picture; 

(c) creating description of actions by each object from the 
region map, the object trajectory ID, the action ID and the 
camera parameter; and 

(d) creating description of scenes by using the description 
of actions by each object. 

A description method for motion picture, the method 
comprising the steps of: 

(a) determining reference plane from the motion picture; 

(b) cutting a region map, an object trajectory ID, an action 
ID and a camera parameter from the motion picture; 

(c) creating description of actions by each object from the 
region map, the object trajectory ID, the action ID and the 
camera parameter; and 

(d) creating description of scenes by using the description 
of actions by each object. 
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A description method for motion picture, the method 
comprising the steps of: 

(a) cutting a region map, an object trajectory ID, an action 
ID and a camera parameter from the motion picture; 

(b) creating description of actions by each object from the 
region map, the object trajectory ID, the action ID and the 
camera parameter; and 

(c) creating description of scenes by using the description 
of actions by each object, 

A computer readable storage medium which has recorded 
management data for searching motion picture, the management 
data comprising of: 

(a) data of description of actions by each object defined by 
position on reference plane and predefined type of actions; 
and 

(b) data of description of scenes defined by the data of 
description of actions. 

A computer readable storage medium which has recorded a 
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program, the program having a computer execute the steps of: 

(a) determining reference plane which represents information 
of object positions included in the motion picture; 

(b) representing changes over time of each object on the 
reference plane as a trajectory; 

(c) setting a description unit based on predefined type of 
actions of each object by using changes in shape of each 
object so as to assign actions of each object as each 
behavioral section; and 

(d) defining each scene by plural objects. 
Description of the Symbols 



401 : 


Video data 


410 : 


Image processing 


420 : 


Data groups 


422 : 


Region map 


424 : 


Obj ect traj ectory ID 


426 : 


Action ID 


428 : 


Camera parameter 


430 : 


Act description 
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442: Reference plane description 
456: lAct description 
4 60: Search engine 
470: User keyword 
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Claims 

1. A description means for contents of motion picture, said 
means comprising of : 

(a) means for setting reference plane; 

(b) means for describing each object on said motion picture 
by position on said reference plane and predefined type of 
actions; and 

(c) means for describing each scene by vising said means for 
describing each object. 

2. A search means for contents of motion picture, said means 
comprising of : 

(a) means for setting reference plane; 

(b) means. for describing each object on said motion picture 
by position on said reference plane and predefined type of 
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actions ; 

(c) means for describing each scene by using said means for 
describing each object; and 

(d) means for searching motion picture by using said means 
for describing each object or said means for describing each 
scene . 

3. A description method for motion picture, said method 
comprising the steps of: 

(a) determining reference plane which represents information 
of object positions included in said motion picture; 

(b) representing changes over time of each object on said 
reference plane as a trajectory; 

(c) setting a description unit based on predefined type of 
actions of each object by using changes in shape of each 
object so as to assign actions of each object as each 
behavioral section; and 

(d) defining each scene by plural objects. 

4. A search method for motion picture, said method 
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comprising the steps of: 

(a) setting reference plane which represents information of 
object positions included in said motion picture; 

(b) representing changes over time of each object on said 
reference plane as a trajectory; 

(c) setting a description unit based on predefined type of 
actions of each object by using changes in shape of each 
object so as to assign actions of each object as each 
behavioral section; 

{d) defining each scene by plural objects; and 

(e) searching a specific scene by using said actions of each 
object or said scene. 

5. A description method for motion picture, said method 
comprising the steps of ; 

(a) determining reference plane from said motion picture; 

(b) cutting a region map, an object trajectory ID, an action 
ID and a camera parameter from said motion picture; 

(c) creating description of actions by each object from said 
region map, said object trajectory ID, said action ID and 
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said camera parameter; and 

(d) creating description of scenes by using said description 
of actions by each object. 

6. A description method for motion picture, said method 
comprising the steps of: 

(a) determining reference plane from said motion picture; 

(b) cutting a region map, an object trajectory ID, an action 
ID and a camera parameter from said motion picture; 

(c) creating description of actions by each object from said 
region map, said object trajectory ID, said action ID and 
said camera parameter; and 

(d) creating description of scenes by using said description 
of actions by each object. 

7. A description method for motion picture, said method 
comprising the steps of : 

(a) cutting a region map, an object trajectory ID, an action 
ID and a camera parameter from said motion picture; 

(b) creating description of actions by each object from said 
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region map, said object trajectory ID, said action ID and 
said camera parameter; and 

(c) creating description of scenes by using said description 
of actions by each object. 

8 . A computer readable storage medium which has recorded 
management data for searching motion picture, said 
management data comprising of: 

(a) data of description of actions by each object defined by 
position on reference plane and predefined type of actions; 
and 

(b) data of description of scenes defined by said data of 
description of actions. 

9 . A computer readable storage medium which has recorded a 
program, said program having a computer execute the steps 
of: 

(a) determining reference plane which represents information 
of object positions included in said motion picture; 

(b) representing changes over time of each object on said 
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reference plane as a trajectory; 

(c) setting a description unit based on predefined type of 
actions of each object by using changes in shape of each 
object so as to assign actions of each object as each 
behavioral section; and 

(d) defining each scene by plural objects. 
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[Table 1] 

Description of Reference plane; 
Reference plane: :^ Model of the ground 
<RefID> text 

<Plane> cordinates — 
<Metric> array of numeric — 



Name of reference plane 

Center of the Area (e.g. (0,0)) 

define transformation (e.g. 3x3 matrix for 
AfFme transformation) 



Description of Zone Description: 

Zone Description: := Define meaningful space on the ground 

<Zone ID> text Identified the zone on the ground 

(e.g. Goal, Center line) 

<Space> space desc define the space on the ground 



Description of Camera Spec (option): 
Camera Spec::= Define camera model 

<CameraType> text Identify camera model 

<Param Array> array of numeric define model transformation 
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[Table 2] 

Description of Action: 

Acrion:: Describe single player's action 

<Action ID> text 

<T-Interval> time interval 

<Object ID> numeric 



Action Symbol (e.g. Run, Kick Walk, etc) 
(text representing types of actions) 

Time Interval of this action (represented 
by starting and ending times) 

Object Identifier (object of this action) 



<Trajectory> Time Stamped Polyline 
(a line v^ith a time attribute of each node) 



• Trajectory of the player in this action 
(trajectory of an object on reference plane 
in the time interval of this action) 
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[Table 3] 

Description of lAction: 

lAction::-^ Meaningful event in the domain, composed of multiple players and ball 

<IAction ID> text Event Symbol 

(e,g. Shot, Pass, Through Pass, etc) 
(text representing types of events) 

<T-Interval> time interval Time interval of this action 

<No of Object> numeric Number of Objects 

<Object ID> array of numeric Array of Objects identifier 

<Space> Trajectory Spatial description of this action 

Polylines, Polygons 
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[Table 4] 

Definition of lAct: 
begin 

Definition of lAct 

A list of an lAct, a child Act and a child ball (a child lAct and a child Act, etc. mean 
an I Act and an Act comprising this I Act) 

where 

[Ail the judgment parts and judgment statements with auxiliary functions are linked 
by AND] 

fill 

[An element assignment part of a defined TAct] 
end 

Here, variable symbols in a define statement have respective types which are 
represented by a first character of each variable. The types are as follows. 



Type Description 

f a fi-ame 

t a time interval (consisting of starting and ending frames) 

o an object 

O a group of objects (consisting of a set of objects and the number of 
them) 

p a point (consisting of spatial component x and y) 

P a group of points (consisting of a set of points and the number of them) 



A a group of points, which means a polyline linking them 

1 a time point (consisting of spatial component x, y and time component t) 

L a group of time points (consisting of a set of time points and the number 

of them) 

i an integer 

d a real number 
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[Table 5] 



Line 


Define slatement 


Description 


I 


begin 


Beginning 


2 


iact Througli_pass tO OO LO 


Here, it defines an Iact "Through Pass". 


3 


child iact 1 Passtl Ol LI 


lact pass as one comprising this Iact. 


4 

3 


child act 3 Slay Walk Run 12 o2 false L2 


And an Act of defense side player 1 (it means that a player with object ID o2 
(Stayed, Walked or Ran) moving on trajectory L2 during time interval t2). 


c 
O 


cni]a_acio May waiK. Run iJ oJ raise L3 


An Act 01 deiense side player 2. 


7 


where 




8 


greater than o2 o3 


Two players of tlie defense side have different object IDs (their IDs are o2, o3). 


9 


get_object_from_GO o4 1 Ol 


Take the first player of Iact ("Pass'*) to put in variable o4. 


10 


not same team o4 o2 


o4 and o2 are different teams. 


1] 


not same team o4 o3 


o4 and o3 are different teams. 


12 


set length_of_polyline dO LI 


Length of a pass is measured. 


13 


less than dO 20.0 


Length of 20 m or les.s. 


14 


temporaloverlap t2 13 


Two players of the defense side have overlapping time intervals 


15 


set_temporal_overlapping_period t4 12 13 


The overlapping time intervals to variable t4. 


16 


temporal overlap tl t4 


There is an overlap in time intervals of t4 and an Iact pass 


17 


set_lemporaI_overlapping__period t5 tl t4 


OverlaoDUiff time intervals fcrtn<:eniientlv averSvtnnirta Ar-tc pnH larft n^cce>c ^ftVi*! 
two players of the defense side) to variable t5. 


18 


get_frame_start_of_period fO t5 


Put a starting fi-ame of time interval tS in time point fO. 


19 


get frarae_end of_period fl t5 


Put an ending frame of time interval t5 in time point fl 


20 


set ST_GL_dividing_poinl pO L2 fO 


The position (location) of player 1 of the defease side at time point fD to variable 
nO 

X puiiiiiuLi ^Luk.^i.iuitj fji ^jiaya ui ulc ucxcnbc aiuc ai iims pomi ikj xo variaoie 
pl. 

The position (location) of player 1 of the defense side at time point fl to variable p2. 


21 


setS_ST_GL_dividing_point pi L3 tD 


22 


set^ST GL_dividingjpoint p2 L2 fl 


23 


set_ST_GL_dividing_point p3 L3 fl 


The position (location) of player 1 of the defense side at time point fl to variable p3. 


24 


get j5oint_from_poly line p4 1 LI 


The starting point of a pass to p4. 


25 


get_point fromjDolyline p5 -1 LI 


The ending point of a pass to p5. 


26 


Create Jine PI 2 p4 p5 


Create a line linking p4 and p5 (a pass course). 


27 


Create line P22p0pl 


A line linking pO and pl (a line Unking the two players of the defense side at time 
point fO). 


28 


Create Jine P3 2 p2 P3 


A line linking p2 and p3 (a line linking the two players of the defense side at time 
point fl ). 


zy 


spatial cross PI P2 


Pl and P2 are crossing. 


30 


set_distance_point_and Jine dl pO p4 p5 


The distance Irom pass course pO at time point fO is sought. 


31 


set distance^point and line d2 pl p4 p5 


The distance from pass course pl at time point fO is sought. 


32 


less than dl 7.0 


The distance is 7 m or less. 


33 


less_than d2 7.0 


Same. 


34 


spatiaI_cros.s P 1 P3 


Pl and P are crossing. 


35 


set diltjiticf' "nrtint anH linp A'K rvd. -nS 


The distance from a pass course at time point fl is sought as above. 


36 


set_distance_point_and Jine d4 p3 p4 p5 




37 


less_than d3 7.0 


The d^tance is 7 m or less. 


38 


less than d4 7.0 




39 


fill 


If the above conditions are met, 


40 


lOtl 


Assign the time interval of tl (Iact pass) to tO (the time interval of an Iact through 
pass). 


41 


OOOl 


Assign the group of objects of Ol (Iact pass) to Ol (the group of objects of an Iact 
through pass). 


42 


LOLl 


Assign LI (a trajectory of Iact pass) to LO (spatial representation of Iact through 
pass). 


43 


end 


End. 
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[Table 6] 

Line Define statement 

1 pass: 

2 begin 

3 lact Pass tO OO LO 

4 Child„actKick|Jump|Slidingtl ol LI 

5 Child act RuniStay|WaIk t2 o2 L2 

6 Ball t3 L3 

7 where 

8 same_team(ol,o2) 

9 get frame start ^period fl t3 

10 get_frame_end_period f2 t3 

11 temporal overlap tl t3 

12 temporal_overlap t2t3 

13 set_ST_GL_dividing_point pi fl L3 

14 set_ST_GL_dividing_point p2 f2 L3 

15 set_ST_GL_dividing__point p3 fl LI 

16 set_ST_GL_dividing_point p4 f2 L2 

17 set_distance_point_to_point dl pi p3 

18 set_distance_point to_point d2 p2 p4 

19 Less_than dl 0.5 

20 Lessjhan d2 0.5 

21 set_Go_from_objects Ol 2 ol o2 

22 fill 

23 tOt3 

24 L0L3 

25 OOOl 

26 End 
27 

28 long pass: 

29 begin 

30 iact Longjpass tO OO LO 

31 childjact Pass tl Ol LI 

32 where 

33 setJength_of_polyline dO LI 

34 Greater Than dl 30.0 

35 fill 

36 tOtl 

37 OOOl 

38 LOLl 

39 end 
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[Table 7] 

Line Define statement 

1 feed pass; 

2 begin 

3 lact Feed Pass tO OO LO 

4 Child Jact Pass tl Ol LI 

5 Child act Run t2 o2 L2 

6 where 

7 get__object_from GO o3 -1 01 

8 same object o2 o3 

9 temporal_during tl t2 

10 fill 

11 tOtl 

12 OOOl 

13 LOLl 

14 end 
15 

16 cross pass: 

17 begin 

18 lact Cross pass to OO LO 

19 Child Jact Pass tl Ol LI 

20 where 

21 get_frame start_period fl tl 

22 get_frame_end_period f2tl 

23 set_ST_GL_dividing_point pi fl LI 

24 set_ST_GL_dividing_j)oint p2 f2 LI 

25 setJength of_polyline dl LI 

26 set__X_distance_point_and_point d2 pi p2 

27 greater_than dl 30.0 

28 less than d2 5.0 

29 fill 

30 to tl 

31 OOOl 

32 LOLl 

33 end 
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[Table 8] 

Line Define statement 



1 gain pass: 

2 begin 

3 lact Gain_j)ass tO OO LO 

4 Childjact Pass tl Ol LI 

5 where 

6 get frame start period fl tl 

7 get_frame_endjperiod f2 tl 

8 set_ST_GL_dividing__point pi fl LI 

9 set_ST_GL„dividing_point p2 f2 LI 

10 setJength_of_Polyline dl LI 

1 1 set_Y_distance_point_andj)oint d2 pi p2 

12 greater Jhan dl 30.0 

13 less_than d2 5.0 

14 fill 

15 tOtl 

16 OOOl 

17 LOLl 

18 end 
19 

20 centering: 

21 begin 

22 lact Cross pass tO OO LO 

23 Childjact Pass tl Ol LI 

24 where 

25 get_frame_start_period fl tl 

26 get_frame_end_period £2tl 

27 set_ST_GL_dividing_point pi fl LI 

28 set_ST_GL_dividing_point p2 f2 LI 

29 setJength_of_Polyline dl LI 

30 greater Jhan dl 8,0 

31 set_X_distance_point_and_point d2 pi p2 

32 less Jhan d2 5.0 

33 spatial j)oint in p2,"Goal Area" 

34 fill 

35 tOtl 

36 OOOl 

37 LOLl 

38 end 
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[Table 9] 

Line Define statement 



1 wall pass: 

2 begin 

3 iact l-2_pass to OOLO 

4 childjact Passtl 01 LI 

5 childjact Pass t2 02 L2 

6 child_act Stay Walk Run t3 ol L3 

7 where 

8 get_object_from_GO o2 1 01 

9 get object from GO o3 -1 01 

10 get_object__fi-om_^GO o4 1 02 

11 get_object_from_GO o5 -1 02 

12 not_same_team ol o2 

13 same object o2 o5 

14 same object o3 o4 

15 set_temporal_distance_period il tl t2 

16 Less Than il 5 

17 not_same_team ol o2 

18 set_temporal_concatination_period t4 tl t2 

1 9 temporal during t3 t4 

20 get_frame_start_ofjperiod fl t4 

21 get_frame_end_of_period f2 tl 

22 get_frame_end_of_period f3 t4 

23 set_ST_GL_dividing_point pi LI fl 

24 set_ST_GL_dividing__point p2 LI f2 

25 set_ST_GL_dividing_point p3 L2 f3 

26 Great e_Area Al 3 pi p2 p3 

27 set_ST_GL_dividingj>oint p4 L3 fl 

28 set ST„GL_dividing_point p5 L3 f3 

29 spatial_point in p4 Al 

30 spatial_pointjn p5 Al 

3 1 set ST_GL_concatinate_polyline L4 LI L2 

32 fill 

33 t0t4 

34 OOOl 

35 L0L4 

36 end 
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[Table 10] 

List of group of auxiliary functions: 
(1) Functions suitable for soccer 

same_team ( PlayerO, Playerl ) 

same team ( Player, Location ) 
It shows whether PlayerO and Playerl belong to the same team. 
One of them may represent a position such as "GOAL." 



(2) Functions that generally hold apart from soccer 





[Temporal] 




1 


get_frame_starl_of_period ( fl ,t 1 ) 


Assign the starting point of time interval tl to fl. 


2 


get_frame_end_of_period (fl.tl) 


Assign the ending point of time interval tl to fl. 


3 


set_period_from_frames (tl,fl,0) 


Create time interval tl from two time points fl, f2. 


4 


get_period_of_GL (tl ,L1 ) 


Assign the lifetime interval of trajectory LI to time interval tl. 


5 


temporal in (fl ,t1 ) 


Time point fl is inside time interval tl. 


6 


temporal meet(tljt2) 


tl and t2 exist in this order and ending Doint of tl and '!tarti'np nnitit 
of t2 are the same. 


7 


temporal overlap (tl,t2) 


There is a time intprval Avfrlannincr time intprval*! tl atiH t? 


8 


temporal start (tl ,12) 


TiTTiP intprvfll<5 tl Jinfl f? Vinvp tlip Qftmp <ststTtii\o rv^ini 


9 


temporal_finish (tl,t2) 


Time intervals tl and t2 have the same ending point. 


10 


temporal_during (tl,t2) 


Time interval tl is completely included in time interval t2. 


11 


temporal_equal (tl ,t2) 


Time intervals tl and t2 have the same starting and ending points. 


12 


temporal_before (t 1 ,t2) 


Time interval tl ends earlier than the starting point of t2. No 
overlapping time interval. 


13 


set temporal overlapping_period 
(tl,t2,t3) 


Assign the overlapping time intervals of t2 and t3 to tl . 


14 


set_temporal_concatmation__period 
(tl,t2,t3) 


Assign the concatenated time intervals of t2 and t3 to tl. 


15 


set temporal distancej3enod 
(il,t2,t3) 


Assign the difference betw^een ending point of time interval t2 and 
starting point of t3 to il. 


16 


set ST GL dividing locus 
(Ll,tl,L2) 


Assign to LI the trajectory of a part applicable to partial interval tl of 
the lifetime interval of trajectory L2. 


17 


set ST GL concatinate locus 
(L1,L2,L3) 


Assign to LI the trajectory concatenating trajectories L2 and L3. 




[Spatial] 




18 


set ST GL dividing_point 

(pULi.n) 


Assign the position of trajectory LI at time point ft) to pi. 


19 


set_point_from_locus (pljl) 


Convert a point 11 of trajectory data to position data pi . 


20 


setjioint (pl,il,i2) 


Define position pi of which x, y coordinates are il, i2. 


21 


Create_Polyline (Pl,il,pl,p2,...) 


Create line PI linking point sets made up of pi, p2, ...(the nimiber, 
il). 
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[Table 11] 

(2) Functions that generally hold apart from soccer (continued) 





rTemnorflll 




22 


Create_Area (Al,il,pl,p2,...) 


Create polyline Al linking point sets made up of pi, p2, ...(the 


23 


set_length_of_polyline (dl,Xl) 


Assign the length of the line shown by XI to dl. XI is P or L. 


24 


setdei stance_point_and_poinl 


Assign the Euchd's distance between positions xl and x2 to dl. x is p 

or i. 


Z_J 


set X distfliicc point fliid point 
{d],xl,x2) 


Assign the distance on axis x between positions xl and x2 to dl. x is 
p or 1. 


26 


set Y distance_point and__point 
(dl,xl,x2) 


Assign the distance on axis y between positions xl and x2 to dl. x is 
p or 1. 


27 


sel_distance_point_and_line 


Assign the distance between position xl and line PI to dl . x is p or 1 . 


28 


act uiouuiuc puiitl oilU /\lCa 

(dl,xUAl) 


/Aaaigii ulc uiaicuiLc uciwcL/ii put>iLiun xi anu pojyime /\i lo cii. x is p 
or 1. 


29 


spatial_point_in (p 1 ,X 1 ) 


Position pi is included in XI. X is P or A. 


30 


spatial_iiiie_in (P1,A1) 


Line PI is completely included in polyline Al . 


31 


spatial_apart (P1J>2) 


No overlap of two lines pi and p2. 


32 


Spatial Jine_touch (P1;K1) 


There is a shared point between PI and XI . X is P or A. 


33 


spatial_through (Pl^l) 


Pi is penetrating polyline Al . 


34 


spatial_overlap (A1,A2) 


Two polylines Al & A2 are partly overlapping. 


35 


spatial_contain (Al^) 


Al is completely included in A2. 


36 




r\l allU r\J, cUC lUUdiliig aL a puilll Ox a line. 


37 


*?natial Hi^imint TAT AO^ 




38 


*?natia1 r.rfi<!'5 /'PI P7^ 


i WiJ LiliCd IT 1 <U.1U ITZ cUC CJ U£>t>lllg. 




[Object handling] 




39 




i. WU UUJCl/Lo Ul OC LfZ die UlC E^CtlllC UUJCt/lb. 


40 


not same object (ol,o2) 


Two objects ol & o2 are different. 


41 


get_object_from_GO (ol,il,01) 


il-th element of set of object Ol is ol. 


42 


set_GO_from_objects 
(Ol,il,oUo2,..0 


Create object set Ol from ol, o2.. . 


43 


get_number_from_GO (il ,Oi ) 


To il seeking the number of elements of object set Ol, 




[Numerical} 




44 


greater_than (xUx2) 


xl > x2, x is f, d, o. 


45 


lessjhan (xl ,x2) 


xl < x2, X is f, d, 0. 


46 


equal (xl,x2) 


xl = x2, X is f, d, 0. 
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NOTE: For fee for processing a non-English application, complete item IV(3). 

NOTE: A non-English oath or declaration in the form provided or approved by the PTO need not be translated. 
37 C.F,R § h69(b), 

FEES 

IV. 

NOTE; See 37 CF.R § 1.2B(a). 

1 . Fees for claims 

□ each independent claim in excess of 3 

(37 CF.R. § i.492(b))-$ 84.00 small entity— $42.00 $ 

□ each claim in excess of 20 

(37 CF.R. § 1.492(c)}— $18.00; small entity— $9.00 $ 

□ multiple dependent claims(s) 

(37 CF.R. § 1.492(d))- $280.00 small entity— $ 140.00 $ 

2. Surcharge fees 

!xX surcharge set forth in 37 CF.R, § 1 .492(e) for accepting 
the declaration later than 30 months after the priority 
date in filing an application in the U.S. as a designated 
office— $130.00; small entity— $55.00 $ ^30.00 



NOTE: The processing fee in the next item 3 below is not subject to a reduction for small entity status. 

3. □ processing fee set forth in 37 CF.R. § 1.492(f) for 
acceptance of an English translation later than 30 

months after the priority date— $130.00 $ 



130.00 



Total fees 

SMALL ENTITY STATUS 

V, a. □ An assertion that this filing is by a small entity 
NOTE: See 37 C.f.R § 1.28(a). 

(check and complete applicable items) 

□ is attached. 

□ was filed on 

□ was made by paying the basic national fee as a small entity. 

□ is being made now by paying the basic national fee as a small entity, 
b. □ A separate refund request accompanies this paper. 
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EXTENSION OF TIME 

(complete (a) or (b), as applicable) 

VI The proceedings herein are for a patent application. Accordingly, the provisions of 37 
C.RR. § 1,1 36(a) apply. 

(a) □ Applicant petitions for an extension of tinne, the fees for which are set out in 

37 C.F.R. § 1.17(a)(1)-(4), for the total number of months checked, out below: 

□ one month $ no.oo $ 55,00 

□ two months $ 400.00 $200.00 

□ three months $ 920,00 $460.00 

□ four months $ 1,440.00 $720.00 

□ five months $ 1,960.00 $980,00 

Fee: $ 

If an additional extension of time is required, please consider this a petition therefor. 

(check and complete the next item, if applicable) 

□ An extension for months has already been secured. The fee paid 

therefor of $ _ is deducted from the total fee due for the total 

months of extension now requested. 

Extension fee due with this request $ 

or 

(b) E Applicant believes that no extension of tenn is required. However, this condi- 

tional petition is being made to provide for the possibility that applicant has 
inadvertently overlooked the need for a petition and fee for extension of time. 

TOTAL FEE DUE 

Vll. The total fee due is: 

Completion fee(s) $ i3Q.qo 

Extension fee (if any) $ 



VIII. 



TOTAL FEE DUE $ ^^^'^^ 
PAYMENT OF FEES 

□ Attached is a D check □ money order in the amount of $ 



£2 Authorization is hereby made to charge the amount of $ 130,00 

IS to Deposit Account No, p^n-nciip ^ 

□ to Credit card as shown on the attached credit card infonnation authoriza- 
tion form PTO-2038. 
WARNING: Credit card information sboufd not be included on this form as It may become public, 

EI Charge any additional fees required by this paper or credit any overpayment 
in the manner authorized above. 

A duplicate of this paper is attached. 
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IX. 

WARNING: 



AUTHORIZATION TO CHARGE ADDITIONAL FEES 



NOTE: 



NOTE. 



NOTE: 



Accurately count claims, especiaily multiple dependant claims, to avoid unexpected high charges 
if extra claims are authorized. 

"A written request may be submitted in an appiication that is an authorization to treat any concurrent 
or future reply, requiring a petition for an extension of time under this paragraph for its timely submission, 
as incorporating a petition for extension of time for the appropriate length of time. An authorization to 
charge all required fees, fees under § 1. 17, or aJ! required extension of time fees will be treated as 
a constructive petition for an extension of time in any concurrent or future reply requiring a petition 
for an extension of time under this paragraph for its timely submission. Submission of the fee set forth 
in § 1.1 7(a)^ will also be treated as a constructive petition for an extension of time in any concurrent 
reply requiring a petition for an extension of time under this paragraph for its timely submission." 37 
C.F.R. § 1,136(a)(3}. 

"Amounts of twenty-five dollars or less will not be returned unless specifically requested within a 
reasonable time, nor will the payer be notified of such amounts; amounts over twenty-five dollars may 
be returned by check or, if requested, by credit to a deposit account" 37 C.F.R. § 1^6(a). 

^ Please charge, in the manner authorized above, the following additional fees that 
may be required by this paper and during the entire pendency of this application: 
KI 37 C.F.R. §§ 1.492(a)(1), 1.492(a)(4) (filing fees) 
13 37 C,F,R. § 1.492(b), (c). and (d) (presentation of extra claims) 
Because additional fees for excess or multiple dependent claims not paid on filing or on later presentation 
must only be paid or these claims cancelled by amendment prior to the expiration of the time' period 
set for response by the PTO in any notice of fee deftciency {37 C.F.R. § 1.16(d)), it might be best not 
to authorize the PTO to charge additionai claim fees, except possibly when dealing with amendments 
after final action. 

S 37 C.F.R. § 1.17 (application processing fees) 

S 37 C.F.R. § 1.17(a)(1)-{5) (extension fees pursuant to § 1.136(a). 

□ 37 C.F.R. § 1.18 (issue fee at or before mailing of Notice of Allowance, 
pursuant to 37 C.F.R. § 1.311(b). 

Where an authorization to charge the issue fee to a deposit account has been filed before the mailing 
of a Notice of Allowance, the issue fee will be automatically charged to the deposit account at the time 
of mailing the notice of allowance. 37 C.F.R. § 1.311(b). 

37 C.F.R. § 1.28(b) requires "Notification of any change in loss of entitlement to small entity status must 
be filed in the application . . . prior to paying, or at the time of paying . . . issue fee. " From the wording 
of 37 C.F.R. § 1.28(b): (a) notification of change of status must be made even if the fee is paid as "other 
than a small entity"" and (b) no notification is required if the change is to another small entity. 

El 37 C.F.R. § 1 .492(e) and/or (f) surchame fees for filing the declaration and/or 
an English translation of an International application^ latep; than 30 months 
from the earliest-claimed prionj^oaat^, 

WARNING: It would be wise to always checf< this l^t Mrthj6rization. 



Reg. No.: 24.622 

Tei. No.: ( 203 ) 259-1800 

Customer No.: 



NOTE: 



NOTE: 



SIGNATURE OF PRACTITijONER 
Clarence A. Green 




(type or print name of practitioner) 
425 Post Road, 



P.O. Address 



Fairfield, Connecticut 06430 
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DECLARATION AND POWER OF ATTORNEY FOR PATENT 

APPLICATION 

English Language Declaration 



As a below named inventor^ I hereby dcdorc tliat: 

My residence, post office address and citi/xnslup arc as stated below next to my name. 

I believe I am tlie original, first cUid sole inventor (if oiily one iiiime is listed below) or iui original, first imd joint 
inventor (if plural names arc listed below) of die subject matter which is claimed and for wluch a paicni is sought on 
the invention entitled: 

Title: METHOD AND DEVICE FOR DESCRIBING VIDEO CONTENTS 
the specification of which 
(check one) 

n is attached hereto. 



X was filed on as United States Application No. 09/890,168 or PCT 

International Application Number PCT/JPOO/00269_ filed on January 20, 2000 
and was an;endcd on (if applicable) 

I hereby stale lluU I Iiavc reviewed and imdcrstand Ihe contents of the abovc-idcntiEcd spccificaiion, including ihc 
claims, as amended by any amendment refen'ed to above. 

I acknowledge tl;e duty to disclose to tJie United States Patent and Trademark Office all information known to me to 
be material to patentabilit>' as defined in Title 37, Code of Federal Regulations, Section 1 .56. 

I hereby claim foreign prioriiy benefits under Title 35, United States Code, Section I19(a)-(d) or Section 365(b) of 
any foreign applicaiion(s) for patent or inventor's certificate, or Section 365(a) of any PCT International Application 
which designated at lc<)$t one country other than the United States^ listed below and liavc also idcniificd below, by 
checking the box, aiiy foreign application for patent or inventor*s certificate or PCT International application liaving 
a filing date before tliar of the application on which priority is claimed. 



Prior Foreijpfn Application(s) 



(Number) (Country) (Day/Month/Ycar Filed) Priority Not Clnimc d 



PCT/JP0O/0O269 


PCT 


20 January 2000 


□ 


11/016554 


Japan 


26 January 1999 


□ 








□ 








□ 
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I hereby claim the benefit under 35 U.S.C. Section 1 19(e) of any United States provisional 
application(s) listed below: 



(Application Serial No.) {Filing Date) 



(Application Serial No.) (Filing Dfiic) 



(Application Serial No,: (Filing Date) 

I hereby claim the benefit under 35 U.S.C. Section 120 of any United States application(s), or 
Section 365(c) of any PCT International Application designating the United States, listed below 
and, insofar as the subject matter of each of the claims of this application is not disclosed in the 
prior United States or PCT International Application in the manner provided by the first 
paragraph of 35 U.S.C. Section 112, I acknowledge the duty to disclose to the United States 
Patent and Trademark Office all information known to me to be material to patentability as 
defined in Title 37, CF.R., Seaion 1.56 which became available between the filing date of the 
prior application and the national or PCT International filing date of this application: 



(Application Serial No.) (Filing Date) (Status) 

(patented, pending, abandoned) 



(Application Serial No.) (Filing Date) (Status) 

(patented, pending, abandoned) 



(Application Serial No.) (Filing Date) (Status) 

(paicntcd, pending, abandoned) 



I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these 
statements were made with the knowledge that willful false statements and the like so made are 
punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the United States 
Code and that such willful false statements may jeopardize the validity of the application or any 
patent issued thereon. 
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POWER OF ATTORNEY: As a named inventor, I hereby appoint the following attorney(s) 
and/or agent(s) to prosecute this application and transact all business in the Patent and 
Trademark Office connected therewith, (list name and registration number) 



Mnnny W. SclKCtcr (31J22) 
John E. I loci (26,279) 
Joseph C. Rcitmoiid. Jr (1?,753; 
Robert T^.Tfcpp (25,933) 
D^i\\i\ r. MoTTis (32.053) 
Marmn Lhidcrwciscr (46 J 34) 
OwIZ.irick (43»303) 
Derek S. Jcnningi (4 1 .473) 



Uuixii C BniZZone (35.0$2) 
Cl^risaophcr A. llugjiw (26.914) 
Louis J. Tcrccllo (33,200) 
Stephen C. Knufninn (29,551) 
Paul J. OUiTAlcdl (37.411) 
LouiiP, Hcrzbcrfi (41.500) 
Doupjns W. Cimtron (31.596) 
E<iwarJ A. f ccuiinglon (32,5SS) 



All attorneys listed under Customer No.: 2512 

Send Correspondence to: 
Customer No.: 2512 



Direct Telephone Calls to: (name and telephone number) 
Reg. No.: (203)259-1800 



Full naiiK ol'wolc or t'lrfii invcnior; 

lojslg ECHIGQ 



Sole Of lirst invemor's si^uture 





2qo\/(c/(0 



Reitidencc: 



714-1 Kaniiyama-cho. Midori-kiu Yokohama-shi, Kanagavva-kcn, Japan uJ V^?c 



CilizcniJlijp: 

Japan 



Post OilRce Acfdrm: 

714-i Kaniiyania-cho. Midori-ku, Yokohama-shi, Kanagawa-kcn, Japan 



l*un iiiunc of second invcnior: 

MasataKUEQKAWA 



Second inventor's signnturc: 



DATE 



j.ooi/ jo/io 



Residence addrc^:s! 



5*16 Tsutsujigaoka, Aoba-ku. Yokolmma-shi. Kanagawa-kcn, Japan uJ P X- 

Cilizciiship: 

Japan 

?osi Office Address 

5-16 Tsutsujigaoka, Aoba-ku, Yokohama-sUi. Kanagawa-keu, Japan 
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Full name of third invisnior: 

JugjiMAEDA 



Third Inv<otoi'*s sigiiaiurt:; DATE 




Residence nddrciiii! 

1-30-9-206 Kaminogc, Sctagaya-ku, Tokvo-co. Japan P 



Cilizea^hip; 

Japaii 



Post Oflic« Address; 

i-30-9-206 Kaminogc, Setagaya-ku, Tokyo-to, Japan 



Full nainc of fourth invcnlor: 

Alberto TOMITA 



Founll iji venter's si giuiurc ! 




DA*i"2 



5-6-2 LTtsukushijyaoka. Aoba-ku,J 



<ohama-shi. Kanagawa-ken, Japan PX- 



Citiiciiship; 
Brazil 



Post Ofllcc Address: 



5-6-2 Utsukushigaoka, Aoba-ku, Yokohaina-slii^ Kanaga^va-k^^ll, Japan 



Full name of llfth inventor: 



Fiilb in V cm or \ signal urcl 



DATE 



Residence ziddnf4<i;.' 



Chi^^^n^^hip; 



Po,*ct OtTicc Address: 



n Check here if additional pages are attached Number of added pages: Page 4 of 4 
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